Postoperative Opioid Prescribing via Rule-Based Guidelines Derived from In-Hospital Consumption: An Assessment of Efficacy Based on Postdischarge Opioid Use.

BACKGROUND: Many institutions have developed operation-specific guidelines for opioid prescribing. These guidelines rarely incorporate in-hospital opioid consumption, which is highly correlated with consumption. We compare outcomes of several patient-centered approaches to prescribing that are derived from in-hospital consumption, including several experimental, rule-based prescribing guidelines and our current institutional guideline. STUDY DESIGN: We performed a retrospective, cohort study of all adults undergoing surgery at a single-academic medical center. Several rule-based guidelines, derived from in-hospital consumption (quantity of opioids consumed within 24 hours of discharge), were used to specify the theoretical quantity of opioid prescribed on discharge. The efficacy of the experimental guidelines was compared with 3 references: an approximation of our institution's tailored prescribing guideline; prescribing all patients the typical quantity of opioids consumed for patients undergoing the same operation; and a representative rule-based, tiered framework. For each scenario, we calculated the penalized residual sum of squares (reflecting the composite deviation from actual patient consumption, with 15% penalty for overprescribing) and the proportion of opioids consumed relative to prescribed. RESULTS: A total of 1,048 patients met inclusion criteria. Mean (SD) and median (interquartile range [IQR]) quantity of opioids consumed within 24 hours of discharge were 11.2 (26.9) morphine milligram equivalents and 0 (0 to 15) morphine milligram equivalents. Median (IQR) postdischarge consumption was 16 (0 to 150) morphine milligram equivalents. Our institutional guideline and the previously validated rule-based guideline outperform alternate approaches, with median (IQR) differences in prescribed vs consumed opioids of 0 (-60 to 27.25) and 37.5 (-37.5 to 37.5), respectively, corresponding to penalized residual sum of squares of 39,817,602 and 38,336,895, respectively. CONCLUSIONS: Rather than relying on fixed quantities for defined operations, rule-based guidelines offer a simple yet effective method for tailoring opioid prescribing to in-hospital consumption.

Mobilizing data during a crisis: Building rapid evidence pipelines using multi-institutional real world data.

The COVID-19 pandemic generated tremendous interest in using real world data (RWD). Many consortia across the public and private sectors formed in 2020 with the goal of rapidly producing high-quality evidence from RWD to guide medical decision-making, public health priorities, and more. Experiences were gathered from five large consortia on rapid multi-institutional evidence generation during the COVID-19 pandemic. Insights have been compiled across five dimensions: consortium composition, governance structure and alignment of priorities, data sharing, data analysis, and evidence dissemination. The purpose of this piece is to offer guidance on building large-scale multi-institutional RWD analysis pipelines for future public health issues. The composition of each consortium was largely influenced by existing collaborations. A central set of priorities for evidence generation guided each consortium, however different approaches to governance emerged. Challenges surrounding limited access to clinical data due to various contributors were overcome in unique ways. While all consortia used different methods to construct and analyze patient cohorts ranging from centralized to federated approaches, all proved effective for generating meaningful real-world evidence. Actionable recommendations for clinical practice and public health agencies were made from translating insights from consortium analyses. Each consortium was successful in rapidly answering questions about COVID-19 diagnosis and treatment despite all taking slightly different approaches to data sharing and analysis. Leveraging RWD, leveraged in a manner that applies scientific rigor and transparency, can complement higher-level evidence and serve as an important adjunct to clinical trials to quickly guide policy and critical care, especially for a pandemic response.

Evaluating capabilities of large language models: Performance of GPT-4 on surgical knowledge assessments.

BACKGROUND: Artificial intelligence has the potential to dramatically alter health care by enhancing how we diagnose and treat disease. One promising artificial intelligence model is ChatGPT, a general-purpose large language model trained by OpenAI. ChatGPT has shown human-level performance on several professional and academic benchmarks. We sought to evaluate its performance on surgical knowledge questions and assess the stability of this performance on repeat queries. METHODS: We evaluated the performance of ChatGPT-4 on questions from the Surgical Council on Resident Education question bank and a second commonly used surgical knowledge assessment, referred to as Data-B. Questions were entered in 2 formats: open-ended and multiple-choice. ChatGPT outputs were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat queries. RESULTS: A total of 167 Surgical Council on Resident Education and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71.3% and 67.9% of multiple choice and 47.9% and 66.1% of open-ended questions for Surgical Council on Resident Education and Data-B, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained nonobvious insights. Common reasons for incorrect responses included inaccurate information in a complex question (n = 16, 36.4%), inaccurate information in a fact-based question (n = 11, 25.0%), and accurate information with circumstantial discrepancy (n = 6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of questions answered incorrectly on the first query; the response accuracy changed for 6/16 (37.5%) questions. CONCLUSION: Consistent with findings in other academic and professional domains, we demonstrate near or above human-level performance of ChatGPT on surgical knowledge questions from 2 widely used question banks. ChatGPT performed better on multiple-choice than open-ended questions, prompting questions regarding its potential for clinical application. Unique to this study, we demonstrate inconsistency in ChatGPT responses on repeat queries. This finding warrants future consideration including efforts at training large language models to provide the safe and consistent responses required for clinical application. Despite near or above human-level performance on question banks and given these observations, it is unclear whether large language models such as ChatGPT are able to safely assist clinicians in providing care.

An Institutional Curriculum for Opioid Prescribing Education: Outcomes From 2017 to 2022.

INTRODUCTION: Prescription opioids, including those prescribed after surgery, have greatly contributed to the US opioid epidemic. Educating opioid prescribers is a crucial component of ensuring the safe use of opioids among surgical patients. METHODS: An annual opioid prescribing education curriculum was implemented among new surgical prescribers at our institution between 2017 and 2022. The curriculum includes a single 75-min session which is comprised of several components: pain medications (dosing, indications, and contraindications); patients at high risk for uncontrolled pain and/or opioid misuse or abuse; patient monitoring and care plans; and state and federal regulations. Participants were asked to complete an opioid knowledge assessment before and after the didactic session. RESULTS: Presession and postsession assessments were completed by 197 (89.6%) prescribers. Across the five studied years, the median presession score was 54.5%. This increased to 63.6% after completion of the curriculum, representing a median relative knowledge increase of 18.2%. The median relative improvement was greatest for preinterns and interns (18.2% for both groups); smaller improvements were observed for postgraduate year 2-5 residents (9.1%) and advanced practice providers (9.1%). On a scale of 1 to 10 (with 5 being comfortable), median (interquartile range) self-reported comfort in prescribing opioids increased from 3 (2-5) before education to 5 (4-6) after education (P < 0.001). CONCLUSIONS: Each year, the curriculum substantially improved provider knowledge of and comfort in opioid prescribing. Despite increased national awareness of the opioid epidemic and increasing institutional initiatives to improve opioid prescribing practices, there was a sustained knowledge and comfort gap among new surgical prescribers. The observed effects of our opioid education curriculum highlight the value of a simple and efficient educational initiative.

Use of Open Science Practices in Surgical Journals.

This cross-sectional study assesses the level of adoption of 5 new tools that promote high quality and transparency in surgical research.

Clinical prediction tool pitfalls and considerations: Data and algorithms.

In recent years, many surgical prediction models have been developed and published to augment surgeon decision-making, predict postoperative patient trajectories, and more. Collectively underlying all of these models is a wide variety of data sources and algorithms. Each data set and algorithm has its unique strengths, weaknesses, and type of prediction task for which it is best suited. The purpose of this piece is to highlight important characteristics of common data sources and algorithms used in surgical prediction model development so that future researchers interested in developing models of their own may be able to critically evaluate them and select the optimal ones for their study.

Comparing Rationale for Opioid Prescribing Decisions after Surgery with Subsequent Patient Consumption: A Survey of the Highest Quartile of Prescribers.

BACKGROUND: Opioid prescribing patterns, including those after surgery, have been implicated as a significant contributor to the US opioid crisis. A plethora of interventions-from nudges to reminders-have been deployed to improve prescribing behavior, but reasons for persistent outlier behavior are often unknown. STUDY DESIGN: Our institution employs multiple prescribing resources and a near real-time, feedback-based intervention to promote appropriate opioid prescribing. Since 2019, an automated system has emailed providers when a prescription exceeds the 75th percentile of typical opioid consumption for a given procedure-as defined by institutional data collection. Emails include population consumption metrics and an optional survey on rationale for prescribing. Responses were analyzed to understand why providers choose to prescribe atypically large discharge opioid prescriptions. We then compared provider prescriptions against patient consumption. RESULTS: During the study period, 10,672 eligible postsurgical patients were discharged; 2,013 prescriptions (29.4% of opioid prescriptions) exceeded our institutional guideline. Surveys were completed by outlier prescribers for 414 (20.6%) encounters. Among patients where both consumption data and prescribing rationale surveys were available, 35.2% did not consume any opioids after discharge and 21.5% consumed <50% of their prescription. Only 93 (39.9%) patients receiving outlier prescriptions were outlier consumers. Most common reasons for prescribing outlier amounts were attending preference (34%) and prescriber analysis of patient characteristics (34%). CONCLUSIONS: The top quartile of opioid prescriptions did not align with, and often far exceeded, patient postdischarge opioid consumption. Providers cite assessment of patient characteristics as a common driver of decision-making, but this did not align with patient usage for approximately 50% of patients.

Evaluating Capabilities of Large Language Models: Performance of GPT4 on Surgical Knowledge Assessments.

Background: Artificial intelligence (AI) has the potential to dramatically alter healthcare by enhancing how we diagnosis and treat disease. One promising AI model is ChatGPT, a large general-purpose language model trained by OpenAI. The chat interface has shown robust, human-level performance on several professional and academic benchmarks. We sought to probe its performance and stability over time on surgical case questions. Methods: We evaluated the performance of ChatGPT-4 on two surgical knowledge assessments: the Surgical Council on Resident Education (SCORE) and a second commonly used knowledge assessment, referred to as Data-B. Questions were entered in two formats: open-ended and multiple choice. ChatGPT output were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat encounters. Results: A total of 167 SCORE and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71% and 68% of multiple-choice SCORE and Data-B questions, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained non-obvious insights. Common reasons for inaccurate responses included: inaccurate information in a complex question (n=16, 36.4%); inaccurate information in fact-based question (n=11, 25.0%); and accurate information with circumstantial discrepancy (n=6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of inaccurate questions; the response accuracy changed for 6/16 questions. Conclusion: Consistent with prior findings, we demonstrate robust near or above human-level performance of ChatGPT within the surgical domain. Unique to this study, we demonstrate a substantial inconsistency in ChatGPT responses with repeat query. This finding warrants future consideration and presents an opportunity to further train these models to provide safe and consistent responses. Without mental and/or conceptual models, it is unclear whether language models such as ChatGPT would be able to safely assist clinicians in providing care.

How surgeons use risk calculators and non-clinical factors for informed consent and shared decision making: A qualitative study.

BACKGROUND: The discussion of risks, benefits, and alternatives to surgery with patients is a defining component of informed consent. As shared-decision making has become central to surgeon-patient communication, risk calculators have emerged as a tool to aid communication and decision-making. To optimize informed consent, it is necessary to understand how surgeons assess and communicate risk, and the role of risk calculators in this process. METHODS: We conducted interviews with 13 surgeons from two institutions to understand how surgeons assess risk, the role of risk calculators in decision-making, and how surgeons approach risk communication during informed consent. We performed a qualitative analysis of interviews based on SRQR guidelines. RESULTS: Our analysis yielded insights regarding (a) the landscape and approach to obtaining surgical consent; (b) detailed perceptions regarding the value and design of assessing and communicating risk; and (c) practical considerations regarding the future of personalized risk communication in decision-making. Above all, we found that non-clinical factors such as health and risk literacy are changing how surgeons assess and communicate risk, which diverges from traditional risk calculators. CONCLUSION: Principally, we found that surgeons incorporate a range of clinical and non-clinical factors to risk stratify patients and determine how to optimally frame and discuss risk with individual patients. We observed that surgeons' perception of risk communication, and the importance of eliciting patient preferences to direct shared-decision making, did not consistently align with patient priorities. This study underscored criticisms of risk calculators and novel decision-aids - which must be addressed prior to greater adoption.

The digital transformation of surgery.

Rapid advances in digital technology and artificial intelligence in recent years have already begun to transform many industries, and are beginning to make headway into healthcare. There is tremendous potential for new digital technologies to improve the care of surgical patients. In this piece, we highlight work being done to advance surgical care using machine learning, computer vision, wearable devices, remote patient monitoring, and virtual and augmented reality. We describe ways these technologies can be used to improve the practice of surgery, and discuss opportunities and challenges to their widespread adoption and use in operating rooms and at the bedside.

Open science practices in research published in surgical journals: A cross-sectional study.

Open science practices are research tools used to improve research quality and transparency. These practices have been used by researchers in various medical fields, though the usage of these practices in the surgical research ecosystem has not been quantified. In this work, we studied the use of open science practices in general surgery journals. Eight of the highest-ranked general surgery journals by SJR2 were chosen and their author guidelines were reviewed. From each journal, 30 articles published between January 1, 2019 and August 11, 2021 were randomly chosen and analyzed. Five open science practices were measured (preprint publication prior to peer-reviewed publication, use of Equator guidelines, study protocol preregistration prior to peer-reviewed publication, published peer review, and public accessibility of data, methods, and/or code). Across all 240 articles, 82 (34%) used one or more open science practices. Articles in the International Journal of Surgery showed greatest use of open science practices, with a mean of 1.6 open science practices compared to 0.36 across the other journals (p<.001). Adoption of open science practices in surgical research remains low, and further work is needed to increase utilization of these tools.

Artificial Intelligence-enabled Decision Support in Surgery: State-of-the-art and Future Directions.

OBJECTIVE: To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting. BACKGROUND: To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown. METHODS: Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. RESULTS: Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (<0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy. CONCLUSIONS: Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

The role of digital technology in surgical home hospital programs.

Home hospital (HH), a care delivery model of providing hospital-grade care to patients in their homes, has become increasingly common in medical settings, though surgical uptake has been limited. HH programs have been shown to be safe and effective in a variety of medical contexts, with increased usage of this care pathway during the COVID-19 pandemic. Though surgical patients have unique clinical considerations, surgical Home Hospital (SHH) programs may have important benefits for this population. Various technologies exist for the delivery of hospital care in the home, such as clinical risk prediction models and remote patient monitoring platforms. Here, we use institutional experiences at Brigham and Women's Hospital (BWH) to discuss the utility of technology in enabling SHH programs and highlight current limitations. Additionally, we comment on the importance of data interoperability, access for all patients, and clinical workflow design in successfully implementing SHH programs.

Quantifying the Prognostic Value of Preoperative Surgeon Intuition: Comparing Surgeon Intuition and Clinical Risk Prediction as Derived from the American College of Surgeons NSQIP Risk Calculator.

BACKGROUND: Surgical risk prediction models traditionally use patient attributes and measures of physiology to generate predictions about postoperative outcomes. However, the surgeon's assessment of the patient may be a valuable predictor, given the surgeon's ability to detect and incorporate factors that existing models cannot capture. We compare the predictive utility of surgeon intuition and a risk calculator derived from the American College of Surgeons (ACS) NSQIP. STUDY DESIGN: From January 10, 2021 to January 9, 2022, surgeons were surveyed immediately before performing surgery to assess their perception of a patient's risk of developing any postoperative complication. Clinical data were abstracted from ACS NSQIP. Both sources of data were independently used to build models to predict the likelihood of a patient experiencing any 30-day postoperative complication as defined by ACS NSQIP. RESULTS: Preoperative surgeon assessment was obtained for 216 patients. NSQIP data were available for 9,182 patients who underwent general surgery (January 1, 2017 to January 9, 2022). A binomial regression model trained on clinical data alone had an area under the receiver operating characteristic curve (AUC) of 0.83 (95% CI 0.80 to 0.85) in predicting any complication. A model trained on only preoperative surgeon intuition had an AUC of 0.70 (95% CI 0.63 to 0.78). A model trained on surgeon intuition and a subset of clinical predictors had an AUC of 0.83 (95% CI 0.77 to 0.89). CONCLUSIONS: Preoperative surgeon intuition alone is an independent predictor of patient outcomes; however, a risk calculator derived from ACS NSQIP is a more robust predictor of postoperative complication. Combining intuition and clinical data did not strengthen prediction.

Appraising the Quality of Development and Reporting in Surgical Prediction Models.

This cross-sectional study uses the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis reporting guideline to assess 120 published studies about surgical prediction models.

Trends in Medical Management of Moderately to Severely Active Ulcerative Colitis: A Nationwide Retrospective Analysis.

BACKGROUND: With an increasing number of therapeutic options available for the management of ulcerative colitis (UC), the variability in treatment and prescribing patterns is not well known. While recent guidelines have provided updates on how these therapeutic options should be used, patterns of long-term use of these drugs over the past 2 decades remain unclear. METHODS: We analyzed a retrospective, nationwide cohort of more than 1.7 million prescriptions for trends in prescribing behaviors and to evaluate practices suggested in guidelines relating to ordering biologics, step-up therapy, and combination therapy. The primary outcome was 30-day steroid-free remission and secondary outcomes included hospitalization, cost, and additional steroid usage. A pipeline was created to identify cohorts of patients under active UC medical management grouped by prescribing strategies to evaluate comparative outcomes between strategies. Cox proportional hazards and multivariate regression models were utilized to assess postexposure outcomes and adjust for confounders. RESULTS: Among 6 major drug categories, we noted major baseline differences in patient characteristics at first exposure corresponding to disease activity. We noted earlier use of biologics in patient trajectories (762 days earlier relative to UC diagnosis, 2018 vs 2008; P < .001) and greater overall use of biologics over time (2.53× more in 2018 vs 2008; P < .00001) . Among biologic-naive patients, adalimumab was associated with slightly lower rates of remission compared with infliximab or vedolizumab (odds ratio, 0.92; P < .005). Comparisons of patients with early biologic initiation to patients who transitioned to biologics from 5-aminosalicylic acid suggest lower steroid consumption for early biologic initiation (-761 mg prednisone; P < .001). Combination thiopurine-biologic therapy was associated with higher odds of remission compared with biologic monotherapy (odds ratio, 1.36; P = .01). CONCLUSIONS: As biologic drugs have become increasingly available for UC management, they have increasingly been used at earlier stages of disease management. Large-scale analyses of prescribing behaviors provide evidence supporting early use of biologics compared with step-up therapy and use of thiopurine and biologic combination therapy.

Population-scale analysis reveals patterns in prescribing trends for ulcerative colitis management. Findings include (1) earlier use of biologics in patient trajectories, (2) associations of step-up therapy with higher corticosteroid exposure, and (3) association of combination therapy with positive patient outcomes.

Machine learning nonresponse adjustment of patient-reported opioid consumption data to enable consumption-informed postoperative opioid prescribing guidelines.

Background: Post-discharge opioid consumption is a crucial patient-reported outcome informing opioid prescribing guidelines, but its collection is resource-intensive and vulnerable to inaccuracy due to nonresponse bias. Methods: We developed a post-discharge text message-to-web survey system for efficient collection of patient-reported pain outcomes. We prospectively recruited surgical patients at Beth Israel Deaconess Medical Center in Boston, Massachusetts from March 2019 through October 2020, sending an SMS link to a secure web survey to quantify opioids consumed after discharge from hospitalization. Patient factors extracted from the electronic health record were tested for nonresponse bias and observable confounding. Following targeted learning-based nonresponse adjustment, procedure-specific opioid consumption quantiles (medians and 75th percentiles) were estimated and compared to a previous telephone-based reference survey. Results: 6553 patients were included. Opioid consumption was measured in 44% of patients (2868), including 21% (1342) through survey response. Characteristics associated with inability to measure opioid consumption included age, tobacco use, and prescribed opioid dose. Among the 10 most common procedures, median consumption was only 36% of the median prescription size; 64% of prescribed opioids were not consumed. Among those procedures, nonresponse adjustment corrected the median opioid consumption by an average of 37% (IQR: 7, 65%) compared to unadjusted estimates, and corrected the 75th percentile by an average of 5% (IQR: 0, 12%). This brought median estimates for 5/10 procedures closer to telephone survey-based consumption estimates, and 75th percentile estimates for 2/10 procedures closer to telephone survey-based estimates. Conclusions: SMS-recruited online surveying can generate reliable opioid consumption estimates after nonresponse adjustment using patient factors recorded in the electronic health record, protecting patients from the risk of inaccurate prescription guidelines.

Association of Postsurgical Opioid Refills for Patients With Risk of Opioid Misuse and Chronic Opioid Use Among Family Members.

Importance: The US health care system is experiencing a sharp increase in opioid-related adverse events and spending, and opioid overprescription may be a key factor in this crisis. Ambient opioid exposure within households is one of the known major dangers of overprescription. Objective: To quantify the association between the postsurgical initiation of prescription opioid use in opioid-naive patients and the subsequent prescription opioid misuse and chronic opioid use among opioid-naive family members. Design, Setting, and Participants: This cohort study was conducted using administrative data from the database of a US commercial insurance provider with more than 35 million covered individuals. Participants included pairs of patients who underwent surgery from January 1, 2008, to December 31, 2016, and their family members within the same household. Data were analyzed from January 1 to November 30, 2018. Exposures: Duration of opioid exposure and refills of opioid prescriptions received by patients after surgery. Main Outcomes and Measures: Risk of opioid misuse and chronic opioid use in family members were calculated using inverse probability weighted Cox proportional hazards regression models. Results: The final cohort included 843 531 pairs of patients and family members. Most pairs included female patients (445 456 [52.8%]) and male family members (442 992 [52.5%]), and a plurality of pairs included patients in the 45 to 54 years age group (249 369 [29.6%]) and family members in the 15 to 24 years age group (313 707 [37.2%]). A total of 3894 opioid misuse events (0.5%) and 7485 chronic opioid use events (0.9%) occurred in family members. In adjusted models, each additional opioid prescription refill for the patient was associated with a 19.2% (95% CI, 14.5%-24.0%) increase in hazard of opioid misuse in family members. The risk of opioid misuse appeared to increase only in households in which the patient obtained refills. Family members in households with any refill had a 32.9% (95% CI, 22.7%-43.8%) increased adjusted hazard of opioid misuse. When patients became chronic opioid users, the hazard ratio for opioid misuse among family members was 2.52 (95% CI, 1.68-3.80), and similar patterns were found for chronic opioid use. Conclusions and Relevance: This cohort study found that opioid exposure was a household risk. Family members of a patient who received opioid prescription refills after surgery had an increased risk of opioid misuse and chronic opioid use.